.com
Hosted by:
Unit testing expertise at your fingertips!
Home | Discuss | Lists

Frequent Debugging

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 248 of xUnit Test Patterns for the latest information.
Also known as: Manual Debugging


Manual debugging is required to determine the cause of most test failures.

Symptoms

A test run results in a test failure or a test error. The output of the Test Runner (page X) is insufficient for us to determine the problem so we have to use an interactive debugger (or print statements sprinkled throughout the code) to determine where things are going wrong.

If this case is an exception, we needn't worry about it but if most test failures require this kind of debugging then we have a case of Frequent Debugging.

Causes

Frequent Debugging is caused by a lack of Defect Localization (see Goals of Test Automation on page X) in our suite of automated tests. The failed tests should tell us what went wrong either through their individual failure messages (see Assertion Message (page X)) or through the pattern of test failures. If they don't:

I've encountered this problem most frequently when I had written the higher level (functional or component) tests but had failed to write all the unit tests for the individual methods. (Some people would call this approach test-first development to distinguish it from test-driven development where every little bit of code is pulled into existence by a failing unit test.)

Frequent Debugging can also be caused by Infrequently Run Tests (see Production Bugs on page X). If we run our tests after every little change we make to the software we can remember what we changed since the last time we ran the tests. That means that when a test fails we don't have to spend a lot of time trouble-shooting the software to discover where the bug is; we know where it is because we remember putting it there!

Impact

Manual debugging is a slow, tedious process. It is easy to overlook subtle indications of the problem and spend many hours tracking down a single logic error. It reduces productivity and makes development schedules much less predictable because a single manual debugging session could extend the time required to develop the software by half a day or more.

Solution Patterns

If we are missing the customer tests for a piece of functionality and manual user testing has revealed a problem not exposed by any automated tests, we probably have a case of Untested Requirement (see Production Bugs). We can ask ourselves what kind of automated test would have prevented the manual debugging session. Better yet, once we have identified the problem, we can write the test that exposes it. Then we can use the failing test to do test-driven bug fixing! If we suspect this to be a widespread problem, we can create a development task to identify and write any additional tests that would be required to fill the gap we just exposed.

Doing true test-driven development is the best way to avoid the circumstances that lead to Frequent Debugging. We should start as close as possible to the skin of the application and do storytest-driven development writing unit tests for individual classes as well as component tests for the collections of related classes to ensure we have good Defect Localization.



Page generated at Wed Feb 09 16:39:50 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved

All Categories
Introductory Narratives
Web Site Instructions
Code Refactorings
Database Patterns
DfT Patterns
External Patterns
Fixture Setup Patterns
Fixture Teardown Patterns
Front Matter
Glossary
Misc
References
Result Verification Patterns
Sidebars
Terminology
Test Double Patterns
Test Organization
Test Refactorings
Test Smells
Test Strategy
Tools
Value Patterns
XUnit Basics
xUnit Members
All "Test Smells"
Code Smells
--Obscure Test
--Conditional Test Logic
--Hard-to-Test Code
--Test Code Duplication
--Test Logic in Production
Behavior Smells
--Assertion Roulette
--Erratic Test
--Fragile Test
--Frequent Debugging
----Manual Debugging
--Manual Intervention
--Slow Tests
Project Smells
--Buggy Tests
--Developers Not Writing Tests
--High Test Maintenance Cost
--Production Bugs